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Summary [Ee cream 


e How we are (and have been) 


— Defining the state of the art 
= Foundational research in assurance technology 


— Pushing the state of the practice 
= Application of research to enable application of emerging 
technologies 
= Unmanned aircraft systems (UAS) missions 


— Developing supporting tools and technologies 


= AdvoCATE (Assurance Case Automation Toolset) 
= Proven application in unmanned aircraft systems (UAS) missions 
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7 iy Research Motivation [Ey cpareaeu 


TECHNOLOGIES 


e High-hazard industries are moving to active safety management 
— Safety management system (SMS) in aviation 
— Need to 
= Unify reasoning about technical aspects of safety 
= Support safety-related decision making 


¢ Goals-based regulation is attractive for novel applications 
— When performance standards are absent 
= Unmanned aircraft systems (UAS), Autonomous systems, ... 
— Increases flexibility for regulated entity 
— Evidence-based assurance > safety case 


Foundational research in languages, methodology, and automation 
support 
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TECHNOLOGIES 


e MIZOPEX (2013) 
— NASA Earth science mission with Sierra UAS off Alaska coast 
— Flight in combination of US National Airspace + Oceanic Airspace 
— Use of air defense radar for detect and avoid 


— Project needed FAA approval through submission of safety case —a 
detailed safety justification 


e UTM (2016 — Ongoing) 
— Fleet of small UAS demonstrating low-altitude traffic management 
system 
— Flight in US national airspace, over sparsely populated land 
— Use of ground-based radar for detect and avoid 
— Project needed FAA approval through submission of safety case 


Practical application of our research solutions 
in response to customer needs 
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TECHNOLOGIES 


‘A Safety case is a structured argument, supported by a body of 
evidence, that provides a compelling, comprehensible and valid case 
that a system is safe for a given application in a given operating 
environment 
- UK MOD, DS-00-56 Issue 4 (2007) 


e Essentially, a safety risk management artifact 
— Other compatible definitions and guidance on content 
= Based on application domain, standard, regulatory paradigm, etc. 

~ FAA: Order 8900.1, FSIMS, vol. 16, UAS 
—~ NAVAIR: Instruction 13034.4 
~ICAO and Eurocontrol: Safety case development manual 
~ Automotive: ISO 26262 
~FDA: Infusion pumps total product lifecycle guidance 
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TECHNOLOGIES 


e FAA (8900.1, FSIMS, vol. 16, UAS) 

— Core content 
= Environment (airspace system) description 
= System description and system change description 
= Airworthiness description of affected items 
= Aircraft capabilities and flight data 
= Accident / incident data 
= Pilot / crew roles and responsibilities 
= Hazard analysis and details of risk analysis, risk assessment, and 

risk control 

= Emergency and contingency procedures 

— Safety risk management plan 
= Hazard tracking and treatment 
= Safety performance monitoring 
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TECHNOLOGIES 


e In general, 
— Explicit statement of safety assurance objectives 
— Heterogeneous evidence 
= Datasheets, design and analysis, verification, operational 
testing,... 
— Structured argument 
= Capturing rationale why evidence supports the claims made 


e Additionally, 
— Safety architecture providing a risk basis 
— Hazard log and hazard analyses 
— Evidence model 
— Monitoring and update 
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STINGER 


; Grama Assurance Cases [ey chara 


TECHNOLOGIES 


‘A documented body of evidence that provides a convincing and valid 
argument that a specified set of critical claims regarding a system’s 
properties are adequately justified for a given application in a given 
environment’ 
- MITRE (2005) 


‘A reasoned and compelling argument, supported by a body of evidence, 
that a system, service, or organization, will operate as intended for a 
defined application, in a defined environment’ 


- Goal Structuring Notation Standard (2017) 


‘A structured set of arguments and a body of evidence showing that an 
(information) system satisfies specific claims with respect to a given 
quality attribute’ 
- National Institute of Standards and Technology (2013) 


Generalization of safety cases to other assurance properties: security, 
dependability, ... 
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Outlook 


System Analysis 
Concept of Operations, 
System/change description, 
Regulations, ... 


proach | EBy charraran 


Assurance Rationale 
(Structured Argument) 


Hazards 


Operational, functional, ... 


Risk Analysis 
and Assessment 


Evidence Artifacts 
Design, Analysis, Verification 
Testing, 


G2 
All identified hazards 
acceptably mitigated 


Assurance claims, 


strategies, context, 


co jument 
operation: 
: 
Likelihood ini Hazard Jui t | 
a8 : : rationale, ... 
- Airsy NI Probable ve 
Our th Probable Fit. = 
GA aircraft nat - 
- phase hazards 


H2 - Stall 


Risk scenarios, design targets, 
risk evaluation 


Operational Evidence 

Verification of safety performance targets 

o <5, Assumption corroboration 

Recovery / Mitigative Hazard tracking, Precursors, ... 
Barriers fi 

Hazard 

Herel ; lose ct Accident / 


Bisat Loss / ' 
Initiatin Control ' ae 7 
Everio State * Harmful | Mitigations 
States or 


States ae Safety requirements 
: Barrier and Control functions 
Prevention / Preventative 
Barriers 


Barrier Modeling - Abstract Safety Architecture 
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Threats / _ 
Causes / 
Initiating 
Events or 
States 


Recovery / Mitigative 
Barriers 


Hazard 


Loss of 
Control 
State 


Prevention / Preventative 


Barriers 


Barrier Modeling - Abstract Safety Architecture 
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Assurance Rationale 
(Structured Argument) 


C2 Gi 
URS ‘Safety of operations 


ci 
CONOPS 
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, Ear Barrier Modeling [ea citer 


e Collection of barrier models providing a risk basis 
— Collection of all factors affecting risk 
— Model for risk qualification/quantification 


Threats / _ 


Causes / Loss of tu / 
Initiating Control OSS 
Events or State -> Harmful 
States States or 
Events 


Prevention Barriers Recovery Barriers 


coeedeacal > Event chain / accident trajectory 


2< Barrier compromise/breach 
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Likelihood: Probable 


Prevention Control (1) 
Barrier Integrity: 0.99 RL: 5B (Low) 
RL: E (Extremely Improbable) 
RS: 5 (Minimal) 
RRL: 5E (Low) 


Recovery Control 
Barrier Integrity: 0.99 
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Hazard 
Avoidance Maneuvers 


Based on the encounter 
geometry, i.e., the location 
of the UA relative to the 
intruder / and a DCP / 
FTP, the RSO directs the 
PIC to initiate an 
appropriate avoidance 
maneuver (divert and land 
immediately, terminate), 
who commands it via the 
GCS. 


Barrier Integrity: 0.99 | 


Airborne UAs 
operating BVLOS 
within the OR 


Independent Flight Abort | 


The PIC invokes an 
independent flight abort 
capability immediately 
shutting off engines and | 


Ground-based Surveillance 


Radar scans the airspace 
and RO monitors the 
surveillance display, to 
detect and track intruder 
heading, altitude, and speed 


Barrier Integrity: 0.99 


Consequence 


halting forward motion 
Barrier Integrity: 0.999 | 


Threat 


between UA and non- 


X 


Non-cooperative 
aircraft intrudes 
into the OR when 


cooperative manned 
aircraft 


IL: B (Probable) 

IS: 1 (Catastrophic) 

IRL: 1B (High) 

RL: E (Extremely Improbable) 
RS: 1 (Catastrophic) 

RRL: 1E (Medium) 


UAs are airborne 


Likelihood: Remote 


IR: 1B (High) —// 
NBR: 1E (Medium), 


Ground-based Surveillance 


RO classifies the intruder as 
an imminent threat if 
separation of intruder 
trajectory from UA location 
and/or designated DCP is | 


Emergency Procedures 


Individual Pilot Actions 


RSO declares an 
emergency, notifies the 
relevant ATC facilities, 
and broadcasts on 
CTAF/UNICOM to notify | 


Pilot of non-cooperative 
aircraft visually acquires 
the UA and takes an 
evasive maneuver | 


[ Barrier Integrity: 0.9 | 


Escalation 
Factor 


intruding aircraft pilot 
Barrier Integrity: 0.5 


projected to be < 1NM 


Barrier Integrity: 0.99 | 


Operations in 
B : & C | inclement 
Loss of voice a rr e r O n t ro iether 
communication cr c ~ 2 
capability Safe nominal Safe nominal 
a x operating operating 
Safe nominal Spectrum Redundancy procedures procedures 
eneranng Management Multiple aviation Continued Operations are 
pioceguie Prior to each flight, all band VHF radios monitoring of conducted in 
All RF frequencies to FF links, including provide weather VMC, when the 
be utilized are verified equipment and redundant voice conditions to stricter weather 
to be free of signals for voice communication ensure that : minimums for 
interference through communication are capability VMG conditions visibility and 
frequency use tested to verify that persist for the cloud ceiling 
approval. A spectrum they are performing duration of flight suitable for VFR 
analyzer deployed as expected, without operations in 
during operations interference Class E airspace 
provides confirmation apply 


that there is no RF 
interference 
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Item of Evidence 
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Safety / Dependability Claims 
‘a ‘N 
‘Ny 
------------> Documentation and Detail 
Chain of Z Developed d Details 
claims e 
reasoning e 
\ 


| 


Undeveloped 
strategy 


Gi 
Root Goal 


Al 
Assumption 


Developed Undeveloped 
sub-goal sub-goal 


Goal Structuring 
Notation (GSN) 
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G1 


LiPo battery 
system failures 
are acceptably 

tolerated 


FMEA of LiPo 
Battery System 


Al 
Independence in failures of 
the primary and the spare 

battery systems 


J1 
LiPo battery system failures 
are characterized by the 
different failure modes 


Usage of 
redundancy 


Show toleration 
over all 
identified 
failure modes 


Battery system Thermal 
short circuits are runaway of the 
eliminated battery packs is 
mitigated 


E1 
Results of 
short circuit 
analysis 


Oct. 30 - 31, 2017 SGT Technology Day. Houston, TX 21 


Motivation 
ASSURANCE CASES 


. STINGER 
Eeamal Tiered Assurance Framework — fSBH ctarranan 
ae lere ssurance Framewo GHAFEARIAN 
Outlook 
Additional 
Tier Core Assurance Concerns and Scope a 
- P Assurance Qualities 
System Safety Due diligence Compliance Processes; 
- Safe concept (safety designed-in) Reduction of risk with Aviation - Maturity, ... 
- Safety in design - ALARP Regulations Input data; 
Safet — Safety in implementation — SFAIRP People; 
Obj a — Safe transition into service — ASARP — Competence, ... 
UNE |) cs Safety in operations Method and Tools; 
— TLOS / Acceptable level of risk — Qualification, ... 
— Safe disposal Safety management system; 
Lifecycle 
All hazards / hazard risk statements, i.e., combination of All applicable Independence of threats; 
1 hazardous situation, hazard release. jeer Effectiveness: 
All relevant consequences across all BTDs. ile iach 
Profile of Risks 
For each hazard, all risk scenarios (consequences), €.g., Coverage (function, environment, 
> midair collision, near midair collision, ground collision, ... interactions, scenarios, ...); 
Specific consequence, e.g., midair collision Independence; 
All causal chains, threats, and dangerous interactions across 
all hazards. 
Individual Risks Depth: 
Specific risk scenario, i.e., causal chain of consequence, Independence; 
3 top event, threats, causes/precursors Proactiveness: Prevention vs. Recovery; 
Applicable system of barriers / safety measures 
Barriers Depth; 
4 Functional safety / fitness for purpose Independence; 
Delivery of required service Common causes/modes, ... 
Controls Reliability and effectiveness; 
eS Functional safety / fitness for purpose Availability; Functional / safety integrity; 
Delivery of required service Resilience; Fail safety; Data integrity; 
Verifiability; ... 
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EXAMPLE Factors Affecting UAS Safety [iay stress. 


« Outlook 


Different configurations 


Diverse environment 
Airborne sensors (Lidar, sonar, 


¢ Populated / urban / built-up areas 
¢ Uncontrolled / controlled airspace 
¢ Low/high density airspace 


FPV camera, Radar) 
Ground sensors (Radar) 
Multiple GCS, Roaming GCS, ... 


Increasing 
complexity in 
mission and 


Varying mission concepts 
Package delivery 
Surveillance 
Aerial inspection 
Mapping, ... 


Varying access profiles 
* Operating range 

¢ Terminal airspace 

¢ Transit (vertical / lateral) 


Combination of operating modes 
¢ Visual line of sight (VLOS) 


¢ Beyond visual line of sight (BVLOS) 
¢ Beyond radio line of sight (BRLOS) 
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e Scope of UAS safety 

— Design assurance 

— Prior to deployment 

— Engineering evidence from development of fitness for purpose 
e Operational assurance 

— Post-deployment, runtime evidence 

— Corroboration of expected safety performance 


e Safety measures should be commensurate with the risk posed by the 
intended operations 


— Level of risk posed dictates safety measures employed and the extent 
of assurance provided 


e Preferred form of safety justification (FAA Order 8900.1) 


— Safety Case 
— Assessment of Acceptable Level of Safety (ALoS) 
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utloo 


System Analysis 


systemhange ose pon, gee 7 
Reguiations, . 2 ; , 
Pd * Primary hazards * Contributory hazards 
Assurance Rationale — PH1: NMAC with non- — CH1: Loss of surveillance 
(Structured Argument) cooperative airborne — CH2: Loss of command 
CR EE ; : entities and control (C2) links 
Pelee ails, ea — PH2: NMAC between UAs — CH3: Loss of ground 
Risk Analysis Teens — PHS: Collision into ground control station (GCS) 
and Assessment ; / structures / people / — CH4: Unrecoverable UA 
besiontaast| —acgurance claims, vehicles failures/malfunction in 
strategies, context, — PH4: Rapid onset of flight 
ravonales inclement weather — CH5: UA deviation from 
— PH5: GPS signal outage approved flight path 
— PH6: UAs exiting the OR and/or exiting the OR 
Riskscenarios,designtargets, =f — CH6: Human factors 
pe ovaluaton, See as, ee * Secondary hazards — CH7: Loss of voice 
Risk Control a 4 PSS — SH1: Lithium fire and/or communication links 
eee explosion 
Safety performance 
; ‘ Mitigations measures, monitors, ... 
, Z.\~——_\ ere 
) I 7 KESTERSON NATIC Safety 
" yMAn WILDLIFE REFUC = off eeurements 
* 7 “tL 2 SR eae | tai Implementation 
: y Oy sees ee : 
wee en VOLTA ¥ a | 4] cone Pg 
eo77 ‘ 


Notional CONOPS a 


Sau Primary and Secondary Hazards 
Baal reas PHI PH2 PH3 PH4 PHS SHI 
% NMAC witha | xmac | Collision into | R#P'd | Gpg | Alkali metal 
. non-cooperative : onsetof | © (Lithium) fire 
. : between | terrain and/or | . Signal 
. aircraft or other in and/or | inclement and/or 
My : UAs __| terrestrial entities Outage 
airspace user weather explosion 
: Conservative choice 
Section 2.2 | MI fie One v ie ¢ 
< 
Section 3.2 | M2q ‘Ground: Daped! v v 
surveillance 
: Measures for 7 
Section 3.1 | M3 ism v v v v 
Section 3.4 ‘Avoidance maneuvers 
and92 | ™ | and contingency procedutes ia # id - i 
Gan Airworthinoés, flight 
Agsitteati MS readinessand crew v v v v 
ppeauen qualification 
Section 6.4 | M6 | _On-béard equipage and ¥ v 
grodnd-safety equipment 
Section9.3 | M7 |, Redundancy 
: 7 Airspace 
OR and portion of Minimal SV covering the augmented TV ill aie deconfliction = - v 
augmented TV above OR * 7 Pre-flight checks, postflight 
Section 6.7 |/M9 | — maintenance and safe v ri 4 # v v 
. : # nominal operations 
Surveillance Requirements a < [+ [~ 
Application management 
AppéndixD | Hazard Analysis Worksheets Table9 | Table 10 | __‘Table 11 Tuble 12 | Table 13 | Table 14 
- 


¢ Avoidance maneuvers, 
Procedures, etc. 

¢ Justification and Rationale 
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Emergency ATC Individual Pilot 
Procedures = Communication ctions 
Excursion from 
the OR 
Pre-mission Gi d-based In-flight Individual Pilot Avoidance Gi \d-based Avoid: Emergency Indepen Individual Pilot 
urveillance ‘ommuncation ions === Maneuvers Surveillance = Maneuvers Flight Abort Actions 
Non-cooperati' 
aircraft, with pilot Non-cooperative 
unaware of UAS aircraft intrudes 
operations, into the OR when 
heading into th UAs are airborne 
TV 


e Residual risk = Consequence probability x severity 
— Probability of disjunction of all paths leading to consequence 
= Inclusion exclusion principle 
— Path probability = Joint probability of all events on path 
= Barrier integrity, threat event probability 
— Assumptions and data 
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EXAMPLE Recall Tiered Assuran (EBB) cuarranian 
ae oe €Ca ere Ssurance GHAFEARIAN 
Outlook 
Additional 
Tier Core Assurance Concerns and Scope a 
- P Assurance Qualities 
System Safety Due diligence Compliance Processes; 
- Safe concept (safety designed-in) Reduction of risk with Aviation - Maturity, ... 
- Safety in design - ALARP Regulations Input data; 
Safet — Safety in implementation — SFAIRP People; 
Obj a — Safe transition into service — ASARP — Competence, ... 
UNE |) cs Safety in operations Method and Tools; 
— TLOS / Acceptable level of risk — Qualification, ... 
— Safe disposal Safety management system; 
Lifecycle 
All hazards / hazard risk statements, i.e., combination of All applicable Independence of threats; 
1 hazardous situation, hazard release. regulatory Effectiveness: 
t 
All relevant consequences across all BTDs. ile iach 
Profile of Risks 
For each hazard, all risk scenarios (consequences), e.g., Coverage (function, environment, 
> midair collision, near midair collision, ground collision, ... interactions, scenarios, ...); 
Specific consequence, e.g., midair collision Independence; 
All causal chains, threats, and dangerous interactions across 
all hazards. 
Individual Risks Depth: 
Specific risk scenario, i.e., causal chain of consequence, Independence; 
3 top event, threats, causes/precursors Proactiveness: Prevention vs. Recovery; 
Applicable system of barriers / safety measures 
Barriers Depth; 
4 Functional safety / fitness for purpose Independence; 
Delivery of required service Common causes/modes, ... 
Controls Reliability and effectiveness; 
eS Functional safety / fitness for purpose Availability; Functional / safety integrity; 
Delivery of required service Resilience; Fail safety; Data integrity; 
Verifiability; ... 
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; Additional 
Tier Core Assurance Concerns and Scope or 
Assurance Qualities 
System Safety Due diligence Compliance Processes; 
— Safe concept (safety designed-in) Reduction of risk with Aviation — Maturity, .. 
— Safety in design - ALARP Regulations nput data; 
Safet - Safety in implementation — SFAIRP People; 
Bike da - Safe transition into service — ASARP — Competence, 
JeCtIVeS | _ Safety in operations Method and Tools; 
— TLOS / Acceptable level of risk — Qualification, ... 
— Safe disposal Safety management system; 
Lifecycle 

Overall Assurance Coverage; 
All hazards / hazard risk statements, i.e., combination of All applicable ndependence of threats; 

1 hazardous situation, hazard release. regulatory Effectiveness: 

requirements 

All relevant consequences across all BTDs. 
Profile of Risks 
For each hazard, all risk scenarios (consequences), e.g., Coverage (function, environment, 
midair collision, near midair collision, ground collision, ... interactions. scenarios ): 
Specific consequence, e.g., midair collision Independence; 
All causal chains, threats, and dangerous interactions across 
all hazards. 
Individual Risks Depth; 

: Specific risk scenario, i.e., causal chain of consequence, Independence; 

3 top event, threats, causes/precursors Proactiveness: Prevention vs. Recovery; 


Applicable system of barriers / safety measures 


Barriers Depth; 
Functional safety Independence; 
Delivery of required service Common causes/modes, ... 
Controls Reliability and effectiveness; 
5 Functional safety / fitness for purpose Availability; Functional / safety integrity; 
: Delivery of required service Resilience; Fail safety; Data integrity; 
Verifiability; 
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C31 G2 
Ground-based surveillance Ground-based wy C1 ' 
system: LSTAR V2 radar, surveillance detects Definition of credible threat: 
ADS-B ground receiver, and tracks airborne Air traffic of the size ofa 
integrated range safety targets that are a single engine Cessna (or 
display (RSD), visual credible threat to UA larger) at the boundary of, or 
observers (VOs) and radar operations sufficiently within, the threat volume (TV) 
operator (RO) early 


c2 
Definition of sufficiently 
early: No less than 90s 
should elapse after 
detection for the intruder to 
arrive at the OR boundary, 
i.e., time to breaching OR 


C16 
Definition of the OR: A 
prismatic volume of 
class G airspace, 
whose base is a 


quadrilateral and Reason over 
height is 1200 ft. AGL. surveillance boundary = 90 seconds 
system 


organization 


G32 
The RSD provides the 
situational picture of the 
airspace of operations and 
its surroundings, consistent 


Gié 

ADS-B ground 
receiver detects and 
tracks UAs operating 


Gi8 
Airborne targets in the 
radar cone of silence that 
pose a credible threat are 
detected and tracked 


G3 


LSTAR V2 radar system within the OR u f 
adequately detects and tracks with reality 
noncooperative/cooperative 
intruder aircraft that can pose a 
credible threat 


C35 
RO is trained and 
qualified under NPR 
7900.3C, to manage 
and interpret 
information from the 
integrated RSD 


Show that the 
RSD provides 
the information 
required for 
situational 
awareness 


Use of 
onboard 
equipage 


Use of visual 
surveillance 


C23 G33 
Pres Reha aa G23 Equipage coe the . G29 ee Ce ee eons a 
a A GPS position reports RSD is capable of tered t id il 
of airspace, of radius 21.5 AUSSI NVEELS Ground-based visual UAs enable their eee by the nee The RSD shows Arsen eau OR, the co ah neh s Sarit 
A in radar cone of observers (VOs) are 4 tt t track: playing a comprehensible view of the 
NM, minus a cone of : ee detection by the Out transponder can be BU acnS augmented TV and the SV OR that is consistent with 
silence of aperture 120 GllemES che will ceployedieuimel agar ground-based ADS-B trusted including position, reality 
degrees immediately visual line of sight location receiver altitude and velocity 
above the radar 


E41 

RSD can 
display the OR, 
ATV and SVina 


E39 
Pre-flight checks for 
surveillance verify 


E40 
Space weather 


E36 
RSD can import 


E23 
At > 4500 ft., threat 


E21 
ES E37 


E24 


aircraft enter the cone fy UAs operating monitoring is 7 
of silence at > 1.28NM Operations eceut BVLOS are Saran ne ES Dec cies KML files showing RSD natively 2D that the RSD display 
i (ian Yer E22 i ; peer e obaicis nieve (ada 3D visualization of shows range of representation ell 
from the radar, which Meterological NASA equipped with an ensure that GPS data including Peta 4 ses oF is calibrated, 
are detected by VOs or Conditions te operating and position reports are position, altitude ne i yom Ys Bs an t eon centered, and : 
are otherwise detected (VMC), suitable qualities functional ADS-B not affected by rare and velocity an yovellal ol including consistent with reality 
earlier by radar for VER flight in NOs areipatt Out transponder normal error pore on a 3D terrain map blanked sectors 
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Detection and 


Barrier Fitness for Purpose 


C31 

Ground-based surveillance 
system: LSTAR V2 radar, 

ADS-B ground receiver, 

integrated range safety 

display (RSD), visual 
observers (VOs) and radar 
operator (RO) 


G2 
Ground-based 
surveillance detects 
and tracks airborne 


q targets that are a 
credible threat to UA 
operations sufficiently, 
early 


C16 


G18 
Airborne targets in the 


Definition of the OR: A 
prismatic volume of 
class G airspace, 
whose base is a 
quadrilateral and 
height is 1200 ft. AGL. 


Reason over 
surveillance 
system 
organization 


ADS-B ground 


C1 
Definition of credible threat: 
Air traffic of the size of a 
> single engine Cessna (or 
larger) at the boundary of, or 
within, the threat volume (TV) 


c2 
Definition of sufficiently 
early: No less than 90s 
should elapse after 
detection for the intruder to 
arrive at the OR boundary, 
ie., time to breaching OR 
boundary = 90 seconds 


The RSD provides the 
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Ground-based surveillance can 
adequately detect and track 


intruders 


Range safety display 


ADS-B 
tracking 


situational picture of the 


radar cone of silence that receiver detects and 
tracks UAs operating 


within the OR 


provides adequate 
situational picture 


G3 
LSTAR V2 radar system 
adequately detects and tracks 


tracking in the 
radar cone of 
silence 


airspace of operations and 
its surroundings, consistent 
with reality 


pose a credible threat are 
detected and tracked 


noncooperative/cooperative 
intruder aircraft that can pose a 
credible threat 


Threats visible 


C23 
Definition of radar SV: A 
3D hemispheric volume 

of airspace, of radius 21.5 

NM, minus a cone of 
silence of aperture 120 
degrees immediately 
above the radar 


E23 
At> 4500 ft., threat 
aircraft enter the cone 
of silence at > 1.28NM 
from the radar, which 
are detected by VOs or 
are otherwise detected 

earlier by radar 
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Use of visual 
surveillance 


G23 
Ground-based visual 
observers (VOs) are 
deployed at the radar 

location 


Airborne threats 

in radar cone of 
silence are within 
visual line of sight 


E24 
Operations occur 


within Visual E22 
Meterological NASA 
Conditions ualified 
(VMC), suitable wes are part 
for VFR flight in of the crew 


VFR / VMC 


Radar detection 
and tracking 


Equipage 


Use of 
onboard 
equipage 


Show that the 
RSD provides 
the information 
required for 
situational 
awareness 


8 G34 
Equipage onboard the 


GPS position reports 


UAs enable their broadcast by the ADS-B The RSD shows Da ta 
detection by the Out transponder can be mer arcetiiacks) 
ground-based ADS-B trusted including position, 


receiver 


displayed 


altitude and velocity 


E40 


operating and 
functional ADS-B 


not affected by ra 
Out transponder 


normal error 


UA minimum 


equipment list 


position reports are 


Eat Space weather E38 RSD lear ort 
UAs cneae monitoring is The RSD receives KML files Areaing 
BULOS TO WLARLE ERED and displays radar 3D visualization of 
equipped with an ensure that GPS 


data including 
position, altitude 

and velocity 
reports 


the OR, augmented 
TV and SV, overlaid 
on a 3D terrain map 


ire 


Range safety 
display functionality 
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RSD is capable of 
displaying the OR, the 
augmented TV and the SV 


E37 
RSD natively 
shows range of 
radar in the form 
of SV including 
blanked sectors 


C35 
RO is trained and 
qualified under NPR 
7900.3C, to manage 
and interpret 
information from the 
integrated RSD 


G33 
The RSD is calibrated and 
centered to provide easily 
comprehensible view of the 
OR that is consistent with 
reality 


Display 
calibration 


E41 
RSD can 
display the OR, 
ATV and SVina 
2D 
representation 


E39 
Pre-flight checks for 
surveillance verify 
that the RSD display 
is calibrated, 
centered, and 
consistent with reality 


Pre-flight checks 
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Tool support 
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Motivation 
Assurance Cases 
Example 

TOOL SUPPORT 
Outlook 


ify . v airs 
> Model Explorer 23 aS 7 
type filter text ° 


>a btSyntax 
ve quasar-sc-eg-14pt 

> wil Project Dependencies 

> [Mlec-example-v2.argument 
lec-example-v2.pdf 
lec.example.argument 
epresentations.aird 
ve quasar-sc-eg-scaling 

> wi Project Dependencies 
lec-example-v2.argument 
lec-example-v2.pdf 
lec.example.argument 
epresentations.aird 


AdvoCATE 


AdvoCATE 


& lec-example-v2 3@ 


27/Q Ol75% BS im 


Assurance 
monitor on 
planner actions 
(TA2) 


C1 
The threshold for 
acceptability is no more 
than 1 unsafe action in 
1E04 operational hours 


Gi 
The probability that the 
LEAP produces unsafe 
actions is acceptably 
low 


C2 
Safety policies for 
the learning-enabled 
automated planner 
(LEAP) 


Jt 
External dependencies 
Capture error 
propagation paths 


$2 S4 


Al $1 Appeal to non- Decomposition / C3 
Ahigh fidelity __ Appeal to violation of all = / over allexternal  / Mission planner 
simulator is used simulation-based applicable safety / component architecture 
verification policies 


dependencies y 


_ LE-CPS platform 
LEC details (TA4) 


2 G3 ae 
Simulation results indicate The LEAP does Functional 
Conditional that the probability of the not violate any Safety Gs 
i 5 it Claim LEC functional Information obtained C6 
ae ale aha eke proce eaeyaa| | (APM) Apenera from mutti-spectral LERS component 
12) poe peed pes pore, (related to LEC data classification is functional 
reliabilit 1M) requirements 
is less than 1E-04 ably; ALIA) trustworthy eq 


wr 7 


$3 


A2 Ei Decomposition over all S7 s8 
The operating evironment LEAP LEAP guarantees traced Show reliability Show robusiness 
model in the simulator is software-in- to the allocated safety fe of multi-spectral / of sensor data 
representative of the actual the-loop policies aa camera / classifier 
operating environment simulation (AQM) y 


results 


= Outline 2 Ble 
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<2 Palette > 
Rhaau-a- 
(& Default oa) 


© Open Hyperlink 


(& Core GSN ca] 


‘® Assumption 


‘— Context 
© Solution 
Goal 


® Justification 


L7 Strategy 
— Is Supported By 
—~ In Context Of 


Developing 


# C5 ss. 
ae . m G7 
w Definition of classification i a 
Design-time fesbie Supervised learning St t d 
verification G4 Adie Wd oleh classifier used for feature ru Cc Uu re 
fic rAA an expected cost Restos 
= ence (TA) LEAP Guarantee: for all plans false positives and false | Reniticaten ane t 
S i in the set of feasible plans, LEC negatives Recall ood rg U m e n Ss 
Mininumialitude= there does not exist an action | contract r 
7a in the set of control actions guarantee TE 
speed = 250 KIAS. such that (commanded altitude |. (TA1) Data used to train the S16. / 
<minimum altitude) and —_—-— te Show using } 
(commanded speed < casetet Wee! ieee model i 
i that is statistically identical / 
ir rte 
minimum speed) piece pe eeu? 
Properties 5% |® | Problems C4) | — 
& Argument lec-example-v2 
| 
id tl Val 
Semantic ain lec-example-v2 _ 
Behaviors Links —+ Is Supported By ISB1, In Context Of ICO1, Is Supported By ISB2, Is Supported By ISB3, In Context Of ICO2, In Context Of IC... 
Documentation Name «= lec-example-v2 
Nodes [=3Goal G1, Strategy $1, Context C1, Goal G2, Solution E1, Context C2, Assumption A1, Assumption A2, Strategy S2, Goal G3.... 


Rulers & Grid 


Appearance | 


Assurance Case Automation Toolset (AdvoCATE) 
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Or BG Qi rr Gros 


AdvoCATE 
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fee Model Explorer 5@ = & oe 


type filter text ix) 


tJ Event FlightAbort 
Event VoiceCommunicationLoss 
v Hazard H1.AirborneUAWithinOR 
Vv +CES 
& CAQ3CES 
v @® Event Instance h1.iNMACLoS 
&& NMACLoS-BcV 
& NMACLoS-BT 
&& NMACLoS-BT-v2 
i Event Instance h1.iORIntrusion 


&, *Surveillance-SliceView 


re 


Ground-based Surveillance 


&, NMACLoS-BcV 


+ QQ 4% Bm 


wa 


[a 


Event Instance h1.iMAC 
Event Instance h1.ilntruderHeadingIntoTV 
Event Instance h1.iUASOperatorsUnaare 
@event Instance h1.iMapMismatch 
(@®eEvent Instance h1.iFlightPathDeviation 
‘vent Instance h1.iORExcursion 
@®event Instance h1.iFlToutsideOR 
6 Event Instance h1.iTerrainSeparationDeteric 
‘vent Instance h1.iInclementWeather 
@event Instance h1.ilnflightLoCc 
e Event Instance h1.iNavigationLoss 
6 Event Instance h1.iAutopilotLoss 
6 Event Instance h1.iUAOtherSubsystemLoss 
Event Instance h1.iC2LinkLoss 
Event Instance h1.iPropulsionLoss 


3 Event Instance h1.iACEmergency 


Control Instance h1.ci1.SeeAndAvoid 


FO monitors the surveillance display and UA track: 
|rorn ADS-B position reports, waming if UA daviate: 
from ths assigned flight paths, altitudes, anc/or 
approaches OR boundary 


ee ae 


HO classites an intruderas a credible 
threat based on radar track and closure 
relative to TV and OR boundaries and UA 
location, If the alrcratt breaches the TV, 
RO notifies RSO of inbound threat aircraft 


& *NMACLOS-BT-v2 5@ 


ect ore fee ED om 


wy 


‘Avoidance Maneuvers 


Bacod on the enootrtor 


“The PIC invokes an 


Independent figtt abort 
‘capabilty inmed ately 
shutting of enges anc 
hing forward econ 


[a 


heading aitude, and speed ie eile 
Barre ntegy: 0.29 Barrer Inieariy: 0.98 er tert: (eee | 
eeRetabe 
[8 Bate op) 
Fcta py 
LC ame 
—-—- - U ____ | fs ectateopnet 
‘Ground-based Surveillance | | Emorgoncy Procedures [individual Pilot Actions ) [82:16 pot) 
SO ceclres an 
resis the 
felvare RTC tacts, 
and broadcasts on 
‘CTAFUNICON 10 retty 
ening areret pict 
Barer rogrty: 05 


UAS operators 
‘are unaware of 
the airspace 


situation 


Radar scans the alrspace and RO 

monitors the surveillance cisplay, to 

detect and track inuderheadng, 
altitude, and speed 


Non-cooperative 
aircraft intruces. 

into the OR when 
UAS are altbome 


Control Instance h1.efci1.WeatherMonitorin 


Event Instance h1.iGCSLoss 
Event Instance h1.iFlightAbort 
i Control Instance h1.efci1.PreflightVMCChe 


RO classities the intruder as an Imminent} 
threat if separation of intruder trajectory 
{from UA location and/or designated DOP 
Is projected to be < 1NM 


o- ; 
n= Outline £3 


6 Event Instance h1.efi.InclementWeather 
Control Instance h1.ci1.InvokeAbort 
Control Instance h1.ci1.DeclareEmergencyh 
Control Instance h1 RSOcommandsDLIo 
Control Instance h1 .ClassifyThreatAfter( 
Control Instance h1.ci1.RadarDetectAndTra 
Control Instance h1 RSOcommandsRTBi 


i 
H 
i 
i 
H 
H 
H 
H 
H 
H 
H 
H 
H 


Automated View 
Extraction 


Bow Tie Modeling 


aid 


E] Properties $2 2) Problems 


Hy 
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@ Event Instance h1.iNMACLoS 
J 
Property 


Semantic 

u Event Instance h1.iINMACLoS 
Behaviors | Associated Argument 
Documentation _ Depth 
Rulers & Grid | eseeietey 
Appearance | Incoming Links 


Initial Likelihood Value 


Initia] Severity 
Name 

Outgoing Links 
Residual Severity 


(@® Event NMACLoS 
+> CES Link, CES Link 


-ATASTROPHIC 


ATASTROPHIC 
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e Hazard analysis and safety requirements capture 


e Structured arguments 
— Pattern specification and automated pattern instantiation 
— Integration of formal methods and formal tool-based evidence 
— Hierarchical and Modular refactoring 
— Argument queries and views 
— Argument verification 
— Metrics 
— Report generation 


e Safety architectures 

— Bow tie modeling 

— Views 

— Transformations (event and barrier split / merge) 
e Evidence management 


e Safety, Mission Assurance, and Risk management (SMART) Dashboard 
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¢ Motivation 


* Tool support 
* OUTLOOK 


e Assurance Cases 
STINGER 
RISC and OHs Sx. 


TECHNOLOGIES 


e NASA adoption of safety case paradigm 


e Promulgated by Office of Safety and Mission Assurance (OSMA) 
— Objective hierarchies (OHs) 
= Decomposition of assurance objectives 
~ Safety, reliability and maintainability, software assurance, range 
safety, ... 
— Risk informed safety case (RISC) 
= System Safety Handbook, vols. 1 & 2 
= Elaborates 
~ NASA acquisition process based on safety performance 
— Supplier requirements for justification of safety performance 
~ Argumentation for rationale capture 
~ Risk assessment and cost-benefit analysis for decision making 
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e Software assurance research program funding (FY 18) 
— Retrospective characterization of assurance afforded by RISC and 
Software OH against an assurance baseline 


— Assurance baseline from NASA ARC BioSentinel mission 
= CFS/CFE 
= V&V artifacts 


= Current NASA assurance standards and guidelines 


— Mapping to RISC and OH to assurance artifacts 
= Analysis of potential gaps and assurance deficits 


— Tool support via AdvoCATE 
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Evang Conclusions and Future Work  [Eay sistemas 


* OUTLOOK 


e Development of end-to-end assurance methodology and tool support 


e Foundational research, informed by and corroborated in practical 
application 


e Safety cases created were the first of their kind 
— MIZOPExX: First civil safety case to be approved 
= NASA Honor Award 


— UTM Safety Case: First civil safety case to be approved for using 
ground-based detect and avoid to conduct BVLOS operations in 
the NAS 
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e Ongoing focus on design-time assurance 
— Artifacts and rationale from development, prior to release-into-service 


e Outlook towards operational assurance through lifecycle 
— In-service safety performance monitoring 


e Dashboard for stakeholder-specific assurance 


e Current focus on safety 
— Expansion in focus to mission assurance 
— Expansion in application domain to spaceflight 
= Initially robotic 
= Eventually, human spaceflight 


Looking for opportunities to infuse our technology 
into other SGT customer projects 
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Abstract 


The Assurance Case approach is being adopted 
in a number of safety-/mission-critical application 
domains in the U.S., e.g., medical devices, 
defense aviation, automotive systems, and, 
lately, civil aviation. This paradigm refocuses 
traditional, process-based approaches to 
assurance on demonstrating explicitly stated 
assurance goals, emphasizing the use 

of structured rationale, and concrete product- 
based evidence as the means for providing 
justified confidence that systems and software 
are fit for purpose in safely achieving mission 
objectives. NASA has also been embracing 
assurance cases through the concepts of Risk 
Informed Safety Cases (RISCs), as documented 
in the NASA System Safety Handbook, and 
Objective Hierarchies (OHs), as put forth by the 
Agency's Office of Safety and Mission Assurance 
(OSMA). This talk will give an overview of the 
work being performed by the SGT team located 
at NASA Ames Research Center, in developing 
technologies and tools to engineer and apply 
assurance cases in customer projects pertaining 
to aviation safety. We elaborate how 

our Assurance Case Automation Toolset 
(AdvoCATE) has not only extended the state-of- 
the-art in assurance case research, but also 


Oct. 30 - 31, 2017 


STINGER 
GHAFFARIAN 
TECHNOLOGIES 


demonstrated its practical utility. We have 
successfully developed safety assurance cases 
for a number of Unmanned Aircraft Systems 
(UAS) operations, which underwent, and passed, 
scrutiny both by the aviation regulator, i.e., the 
FAA, as well as the applicable NASA boards for 
airworthiness and flight safety, flight readiness, 
and mission readiness. We discuss our efforts in 
expanding AdvoCATE capabilities to support 
RISCs and OHs under a project recently funded 
by OSMA under its Software Assurance 
Research Program. Finally, we speculate on the 
applicability of our innovations beyond aviation 
safety to such endeavors as robotic, and human 
spaceflight. 
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